Evaluation Metrics for Generation

Certain generation applications may profit from the use of stochastic methods. In developing stochastic methods, it is crucial to be able to quickly assess the relative merits of different approaches or models. In this paper, we present several types of intrinsic (system internal) metrics which we have used for baseline quantitative assessment. This quantitative assessment should then be augmented to a fuller evaluation that examines qualitative aspects. To this end, we describe an experiment that tests correlation between the quantitative metrics and human qualitative judgment. The experiment confirms that intrinsic metrics cannot replace human evaluation , but some correlate significantly with human judgments of quality and understandability and can be used for evaluation during development. 1 Introduction For many applications in natural language generation (NLG), the range of linguistic expressions that must be generated is quite restricted, and a grammar for a surface realization component can be fully specified by hand. Moreover, iLL inany cases it is very important not to deviate from very specific output in generation (e.g., maritime weather reports), in which case hand-crafted grammars give excellent control. In these cases, evaluations of the generator that rely on human judgments (Lester and Porter, I997) or on human annotation of the test corpora (Kukich, 1983) are quite sufficient .... However. in other NLG applications the variety of the output is much larger, and the demands on the quality of the output are solnewhat less stringent. A typical example is NLG in the context of (interlingua-or transfer-based) inachine translation. Another reason for relaxing the quality of the output may be that not enough time is available to develop a full gramnlar for a new target, language in NLG. ILL all these cases, stochastic methods provide an alternative to hand-crafted approaches to NLG. 1 To our knowledge, the first to use stochastic techniques in an NLG realization module were Langkilde and Knight (1998a) and (~998b) (see also (Langk-ilde, 2000)). As is the case for stochastic approaches in natural language understanding, the research and development itself requires an effective intrinsic metric in order to be able to evaluate progress. In this paper, we discuss several evaluation metrics that we are using during the development of FERGUS (Flexible Empiricist/Rationalist Generation Using Syntax). FERCUS, a realization module, follows Knight and Langkilde's seminal work in using an n-gram language model, but we augment it with a tree-based stochastic model and a lexicalized syntactic grammar. The …